System and method for supporting automatic disabling of degraded links in an infiniband (IB) network

ABSTRACT

A system and method can support automatic disabling of degraded links in an InfiniBand (IB) network. At least one node in a fabric can monitor one or more local ports of the at least one node for one or more error states associated with a link at the at least one node, wherein the link is connected to a local port of the at least one node. The at least one node further allows a subnet manager to observe the one or more error states associated with the link at the at least one node, and allows the subnet manager to set the link in a basic state if the observed error states exceed a threshold. In this basic state, the link allows only SMP traffic and prevents data traffic and non-SMP based management traffic.

CLAIM OF PRIORITY

This application claims the benefit of priority on U.S. ProvisionalPatent Application No. 61/493,330, entitled “STATEFUL SUBNET MANAGERFAILOVER IN A MIDDLEWARE MACHINE ENVIRONMENT” filed Jun. 3, 2011, whichapplication is herein incorporated by reference.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF INVENTION

The present invention is generally related to computer systems, and isparticularly related to supporting an InfiniBand (IB) network.

BACKGROUND

The interconnection network plays a beneficial role in the nextgeneration of super computers, clusters, and data centers. Highperformance network technology, such as the InfiniBand (IB) technology,is replacing proprietary or low-performance solutions in the highperformance computing domain, where high bandwidth and low latency arethe key requirements. For example, IB installations are used insupercomputers such as Los Alamos National Laboratory's Roadrunner,Texas Advanced Computing Center's Ranger, and ForschungszcntrumJuelich's JuRoPa.

IB was first standardized in October 2000 as a merge of two oldertechnologies called Future I/O and Next Generation I/O. Due to its lowlatency, high bandwidth, and efficient utilization of host-sideprocessing resources, it has been gaining acceptance within the HighPerformance Computing (HPC) community as a solution to build large andscalable computer clusters. The de facto system software for IB isOpenFabrics Enterprise Distribution (OFED), which is developed bydedicated professionals and maintained by the OpenFabrics Alliance. OFEDis open source and is available for both GNU/Linux and MicrosoftWindows.

SUMMARY

Described herein is a system and method that can support automaticdisabling of degraded links in an InfiniBand (IB) network. At least onenode in a fabric can monitor one or more local ports of the at least onenode for one or more error states associated with a link at the at leastone node, wherein the link is connected to a local port of the at leastone node. The at least one node further allows a subnet manager toobserve the one or more error states associated with the link at the atleast one node, and allows the subnet manager to set the link in a basicstate if the observed error states exceed a threshold. In this basicstate, the link allows only SMP traffic and prevents data traffic andnon-SMP based management traffic.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 shows an illustration of a fabric model in a middlewareenvironment in accordance with an embodiment of the invention.

FIG. 2 shows an illustration of supporting automatic disabling ofdegraded links in a middleware environment in accordance with anembodiment of the invention.

FIG. 3 illustrates an exemplary flow chart for alleviating networkinstability in a middleware environment in accordance with an embodimentof the invention.

DETAILED DESCRIPTION

Described herein is a system and method that can support automaticdisabling of degraded links in an interconnected network, such as anInfiniBand (IB) network.

FIG. 1 shows an illustration of a fabric model in a middlewareenvironment in accordance with an embodiment of the invention. As shownin FIG. 1, an interconnected network, or a fabric 100, can includeswitches 101-103, bridges and routers 104, host channel adapters (HCAs)105-106 and designated management hosts 107. Additionally, the fabriccan include, or be connected to, one or more hosts 108 that are notdesignated management hosts.

The designated management hosts 107 can be installed with HCAs 105, 106,a network software stack and relevant management software in order toperform network management tasks. Furthermore, firmware and managementsoftware can be deployed on the switches 101-103, and the bridges androuters 104 to direct traffic flow in the fabric. Here, the host HCAdrivers, OS and Hypervisors on hosts 108 that are not designatedmanagement hosts may be considered outside the scope of the fabric froma management perspective.

The fabric 100 can be in a single media type, e.g. an IB only fabric,and be fully connected. The physical connectivity in the fabric ensuresin-band connectivity between any fabric components in the non-degradedscenarios. Alternatively, the fabric can be configured to includeEthernet (Enet) connectivity outside gateway (GW) external ports on agateway 109. Additionally, it is also possible to have independentfabrics operating in parallel as part of a larger system. For example,the different fabrics can only be indirectly connected via differentHCAs or HCA ports.

InfiniBand (IB) Architecture

IB architecture is a serial point-to-point technology. Each of the IBnetworks, or subnets, can include a set of hosts interconnected usingswitches and point-to-point links. A single subnet can be scalable tomore than ten-thousand nodes and two or more subnets can beinterconnected using an IB router. The hosts and switches within asubnet are addressed using local identifiers (LIDs), e.g. a singlesubnet may be limited to 49151 unicast addresses.

An IB subnet can employ at least one subnet manager (SM) which isresponsible for initializing and starting up the sub-net including theconfiguration of all the IB ports residing on switches, routers and hostchannel adapters (HCAs) in the subset. The SM's responsibility alsoincludes routing table calculation and deployment. Routing of thenetwork aims at obtaining full connectivity, deadlock freedom, and loadbalancing between all source and destination pairs. Routing tables canbe calculated at network initialization time and this process can berepeated whenever the topology changes in order to update the routingtables and ensure optimal performance.

At the time of initialization, the SM starts in the discovering phasewhere the SM does a sweep of the network in order to discover allswitches and hosts. During the discovering phase, the SM may alsodiscover any other SMs present and negotiate who should be the masterSM. When the discovering phase is completed, the SM can enter a masterphase. In the master phase, the SM proceeds with LID assignment, switchconfiguration, routing table calculations and deployment, and portconfiguration. At this point, the subnet is up and ready to use.

After the subnet is configured, the SM can monitor the network forchanges (e.g. a link goes down, a device is added, or a link isremoved). If a change is detected during the monitoring process, amessage (e.g. a trap) can be forwarded to the SM and the SM canreconfigure the network. Part of the reconfiguration process, or a heavysweep process, is the rerouting of the network which can be performed inorder to guarantee full connectivity, deadlock freedom, and proper loadbalancing between all source and destination pairs.

The HCAs in an IB network can communicate with each other using queuepairs (QPs). A QP is created during the communication setup, and a setof initial attributes such as QP number, HCA port, destination LID,queue sizes, and transport service are supplied. On the other hand, theQP associated with the HCAs in a communication is destroyed when thecommunication is over. An HCA can handle many QPs, each QP consists of apair of queues, a send queue (SQ) and a receive queue (RQ). There is onesuch pair present at each end-node that is participating in thecommunication. The send queue holds work requests to be transferred tothe remote node, while the receive queue holds information on what to dowith the data received from the remote node. In addition to the QPs,each HCA can have one or more completion queues (CQs) that areassociated with a set of send and receive queues. The CQ holdscompletion notifications for the work requests posted to the send andreceive queue.

The IB architecture is a flexible architecture. Configuring andmaintaining an IB subnet can be carried out via special in-band subnetmanagement packets (SMPs). The functionalities of a SM can, inprinciple, be implemented from any node in the IB subnet. Each end-portin the IB subnet can have an associated subnet management agent (SMA)that is responsible for handling SMP based request packets that aredirected to it. In the IB architecture, a same port can represent a SMinstance or other software component that uses SMP based communication.Thus, only a well defined sub-set of SMP operations can be handled bythe SMA.

SMPs use dedicated packet buffer resources in the fabric, e.g. a specialvirtual lane (VL15) that is not flow-controlled (i.e. SMP packets may bedropped in the case of buffer overflow. Also, SMPs can use either therouting that the SM sets up based on end-port Local Identifiers (LIDs),or SMPs can use direct routes where the route is fully defined by thesender and embedded in the packet. Using direct routes, the packet'spath goes through the fabric in terms of an ordered sequence of portnumbers on HCAs and switches.

The SM can monitor the network for changes using SMAs that are presentedin every switch and/or every HCA. The SMAs communicate changes, such asnew connections, disconnections, and port state change, to the SM usingtraps and notices. A trap is a message sent to alert end-nodes about acertain event. A trap can contain a notice attribute with the detailsdescribing the event. Different traps can be defined for differentevents. In order to reduce the unnecessary distribution of traps, IBapplies an event forwarding mechanism where end-nodes are required toexplicitly subscribe to the traps they want to be informed about.

The subnet administrator (SA) is a subnet database associated with themaster SM to store different information about a subnet. Thecommunication with the SA can help the end-node to establish a QP bysending a general service management datagram (MAD) through a designatedQP, .e.g. QP1. Both sender and receiver require information such assource/destination LIDs, service level (SL), maximum transmission unit(MTU), etc. to establish communication via a QP. This information can beretrieved from a data structure known as a path record that is providedby the SA. In order to obtain a path record, the end-node can perform apath record query to the SA, e.g. using the SubnAdmGet/SubnAdmGetableoperation. Then, the SA can return the requested path records to theend-node.

The IB architecture provides partitions as a way to define which IBend-ports should be allowed to communicate with other IB end-ports.Partitioning is defined for all non-SMP packets on the IB fabric. Theuse of partitions other than the default partition is optional. Thepartition of a packet can be defined by a 16 bit P_Key that consists ofa 15 bit partition number and a single bit member type (full orlimited).

The partition membership of a host port, or an HCA port, can be based onthe premise that the SM sets up the P_Key table of the port with P_Keyvalues that corresponds to the current partition membership policy forthat host. In order to compensate for the possibility that the host maynot be fully trusted, the IB architecture also defines that switch portscan optionally be set up to do partition enforcement. Hence, the P_Keytables of switch ports that connect to host ports can then be set up toreflect the same partitions as the host port is supposed to be a memberof (i.e. in essence equivalent to switch enforced VLAN control inEthernet LANs).

Since the IB architecture allows full in-band configuration andmaintenance of an IB subnet via SMPs, the SMPs themselves are notsubject to any partition membership restrictions. Thus, in order toavoid the possibility that any rough or compromised node on the IBfabric is able to define an arbitrary fabric configuration (includingpartition membership), other protection mechanisms are needed.

M_Keys can be used as the basic protection/security mechanism in the IBarchitecture for SMP access. An M_Key is a 64 bit value that can beassociated individually with each individual node in the IB subnet, andwhere incoming SMP operations may be accepted or rejected by the targetnode depending on whether the SMP includes the correct M_Key value (i.e.unlike P_Keys, the ability to specify the correct M_Key value—like apassword—represents the access control).

By using an out-of-band method for defining M_Keys associated withswitches, it is possible to ensure that no host node is able to set upany switch configuration, including partition membership for the localswitch port. Thus, an M_Key value is defined when the switch IB linksbecomes operational. Hence, as long as the M_Key value is notcompromised or “guessed” and the switch out-of-band access is secure andrestricted to authorized fabric administrators, the fabric is secure.

Furthermore, the M_Key enforcement policy can be set up to allowread-only SMP access for all local state information except the currentM_Key value. Thus, it is possible to protect the switch based fabricfrom un-authorized (re-)configuration, and still allow host based toolsto perform discovery and diagnostic operations.

The flexibility provided by the IB architecture allows theadministrators of IB fabrics/subnets, e.g. HPC clusters, to decidewhether to use embedded SM instances on one or more switches in thefabric and/or set up one or more hosts on the IB fabric to perform theSM function. Also, since the wire protocol defined by the SMPs used bythe SMs is available through APIs, different tools and are commands canbe implemented based on use of such SMPs for discovery, diagnostics andcontrol independently of any current Subnet Manager operation.

From a security perspective, the flexibility of IB architectureindicates that there is no fundamental difference between root access tothe various hosts connected to the IB fabric and the root accessallowing access to the IB fabric configuration. This is fine for systemsthat are physically secure and stable. However, this can be problematicfor system configurations where different hosts on the IB fabric arecontrolled by different system administrators, and where such hostsshould be logically isolated from each other on the IB fabric.

Automatic Disabling of Degraded Links

When IB links are disabled due to excessive error rates, it is difficultto observe the current error rates of the link, or to perform additionaltesting to further diagnose or characterize the problem. If a repairaction is performed to correct the problem, e.g. by replacing a cable orcorrecting the seating of a cable connector, then the link needs to beenabled before it can be tested, in which case the subnet manager mayalso enable/use the link for normal data traffic.

Automated logic can be used to determine whether there are excessiveerror rates. This automated logic also can disable the link when thereare excessive error rates. Since the link has been completely disabled,it is no longer possible to use the link for basic connectivity such assupporting management operations between IB nodes. However, the link maystill be capable of being used for supporting management operationsbetween IB nodes, even when a link is not reliable enough for normaldata traffic. Additionally, there may be no corresponding automatedoperation to enable the link again as a result of detecting asignificantly lower error rate over a significant period of time.

In accordance with an embodiment of the invention, the fabric can ensurethat severely degraded links are not used for data traffic. The fabricallows for the definition of basic policies for automatic disabling ofdegraded links based on associated error rate thresholds. The fabricalso allows for the specification of policies to automatically definedegraded links that are in a non-routable state and to have the SMautomatically observe this state.

Furthermore, subnet level error reporting can be supported in thefabric. The subnet level error reporting can be beneficial in terms ofensuring the states that the SM monitors are coherent, or consistent.The SM can be aware of the error state, and/or the explicit disabledstate, that are associated with the links or with the pair of ports thateach link represents.

Additionally, a local daemon can take the problem link out of normalservice as soon as possible without permanently disabling it. Forexample, the local daemon can reset the link, instead of disabling thelink, or marking the link as a bad link and leaving it up to the SM totake further action (i.e. potentially with longer reaction time). Byresetting the link, the local daemon allows the normal data traffic tobe disabled right away, and there may not be any delay for waiting forthe SM to change the link state. Thus, resetting the link, rather thandisabling the link, can bring the link to the same basic state as the SMcan use initially. Furthermore, the SM can request the link to beenabled again and then keep the link in a basic state or at least notallow the link to be used for data traffic, even in the case where thelocal daemon disables the link. Here, the disabling of a link may removethe last link between the SM and the relevant node (where the daemonoperates), in which case the SM may not be able to request for theenabling of the link.

FIG. 2 shows an illustration of supporting automatic disabling ofdegraded links in a middleware environment in accordance with anembodiment of the invention. As shown in FIG. 2, a fabric 200 includes aSM 202. At least one node 201 in the fabric 200 can have one or moreports 211-213, each of which is associated with one or more links221-223. In the example as shown in FIG. 2, a link 222 associated withthe port 212 on the node 201 is degraded.

A daemon 203 can be used on the node 201 to constantly monitor thesymbol error and other error conditions associated with local linksassociated with the one or more ports 211-213. The node can perform adisable operation on the link if the various error rates exceed aconfigurable threshold during a configurable time interval. Such adisable operation can be reported to the SM 202, e.g. via conventionalsystem management interfaces. A disable error state 209 can beexplicitly recorded in a local stable storage 207, so that the link canavoid being unconditionally enabled again following a reset of the node201. In one example, a configurable policy can be defined to remove thedisable error state 209 following a reset of the node 201.

Instead of performing a local unconditional disable operation for thelocal port/link, the error state 209 can be recorded and made availableto the SM 202 via subnet management traps, or be observable via SMPbased methods, e.g. via extending the set of SMA attributes associatedwith the port. The SM 202 can observe the port error states using thedesignated SMP methods, e.g. using an extension to the normal subnetsweep operations. Also, the SM 202 can observe the port error state as aresult of receiving the corresponding SMPs.

When the SM 202 detects a port 212 with an excessive error rateattribute set, the SM 202 can consider the corresponding link 222 as notoperational in terms of being used by normal data traffic. The SM canstill use this link for further discovery and other subnet managementoperations. However, the SM 202 is prevented from performing furtherdiscovery beyond the remote side of the relevant link in the case of anon-operational link, a not fully responsive remote SMA, or an unknownremote M_Key.

Furthermore, the SM 202 can leave/set the link 222 in a basic state,which allows SMP traffic and prevents both data traffic and non-SMPbased management traffic. In this case, the link 222 can be tested usingSMP based traffic in addition to be used for SMP based managementoperations. Testing can be initiated by the SM 202, or can be carriedout by daemons 203 associated with the involved nodes, or by anothercentralized management entity.

Additionally, the SM 202 can enable the link 222 for normal data trafficfor certain specific purposes. For example, non-SMP based traffic can beused to achieve higher levels of load/stress of the link for providingmore elaborate stress testing of the link 222. Also, the link 222 can beused for other non-SMP based management traffic, such as the managementtraffic using general management packet (GMP) type MADs or higher levelprotocols like internet protocol over InfiniBand (IPoIB), in order tofacilitate generic communication between any management entitiesassociated with the various nodes. In these cases, the SM 202 may notinclude the link in the set of links through which normal data trafficis routed.

When the link is physically down and then comes back again, e.g. due toa cable replacement, the SM 202 can require a specific test procedure tobe carried out for the link before the SM can fully include the link inthe subnet. Such testing can be implemented by per node daemons 203 orby the SM 202 or by another centralized management entity. Then, thetest procedure can be coordinated via additional SMP based methods andattributes associated with each port. When further testing shows asignificantly improved error rate, the port attribute indicatingexcessive errors can be reset, and the SM 202 can then again include thelink in the subnet topology for normal data traffic without anyconstraints.

FIG. 3 illustrates an exemplary flow chart for alleviating networkinstability in a middleware environment in accordance with an embodimentof the invention. As shown in FIG. 3, at step 301, at least one node ina fabric can monitor one or more local ports of the at least one nodefor one or more error states associated with a link at the at least onenode, wherein the link is connected to a local port of the at least onenode. Then, at step 302, a subnet manager can observe the one or moreerror states associated with the link at the at least one node. Finally,at step 303, the subnet manager can set the link to be in a basic stateif the observed error states exceed a threshold.

The present invention may be conveniently implemented using one or moreconventional general purpose or specialized digital computer, computingdevice, machine, or microprocessor, including one or more processors,memory and/or computer readable storage media programmed according tothe teachings of the present disclosure. Appropriate software coding canreadily be prepared by skilled programmers based on the teachings of thepresent disclosure, as will be apparent to those skilled in the softwareart.

In some embodiments, the present invention includes a computer programproduct which is a storage medium or computer readable medium (media)having instructions stored thereon/in which can be used to program acomputer to perform any of the processes of the present invention. Thestorage medium can include, but is not limited to, any type of diskincluding floppy disks, optical discs, DVD, CD-ROMs, microdrive, andmagneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flashmemory devices, magnetic or optical cards, nanosystems (includingmolecular memory ICs), or any type of media or device suitable forstoring instructions and/or data.

The foregoing description of the present invention has been provided forthe purposes of illustration and description. It is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Many modifications and variations will be apparent to the practitionerskilled in the art. The embodiments were chosen and described in orderto best explain the principles of the invention and its practicalapplication, thereby enabling others skilled in the art to understandthe invention for various embodiments and with various modificationsthat are suited to the particular use contemplated. It is intended thatthe scope of the invention be defined by the following claims and theirequivalence.

What is claimed is:
 1. A method for supporting automatic disabling ofdegraded links in a fabric network that includes a subnet manager (SM)and a plurality of network nodes, the method comprising: monitoring, viaa daemon process on a first network node in the fabric network, a localport of the first network node for one or more errors, wherein the oneor more errors are associated with a link that connects the firstnetwork node and a second network node in the fabric network, whereinthe link is in a first state and is configured to transfer both normaldata and subnet management packet (SMP) data; in response to the one ormore errors exceeding a configurable threshold, performing, by thedaemon process, a disabling operation on the link to put the link into asecond state, wherein the link in the second state is configured totransfer only SMP data; recording, by the daemon process, the state ofthe link in a persistent storage, wherein the recorded state isconfigured to prevent the link from being automatically enabled when thefirst network node is reset; and sending, via the link, one or more SMPmessages to the subnet manager, wherein the one or more management datamessages indicate that the link is disabled; wherein the subnet manager,upon receiving the one or more management data messages, operates to usethe link to perform further discovery in the second network node, invokethe daemon process to initiate a test on the link, determine that errorsat the local port of the first network node have dropped below theconfigurable threshold, and change the link from the second state to thefirst state; and wherein the daemon process, in response to the statechange of the link from the second state to the first state, operates toremove the recorded state from the persistence storage when the firstnetwork node is reset.
 2. The method according to claim 1, furthercomprising: requesting, via the subnet manager, the disabled link to beenabled.
 3. The method according to claim 1, further comprising:observing, via the subnet manager, the one or more errors using subnetmanagement packet (SMP) methods.
 4. The method according to claim 1,further comprising: including, via the subnet manager, the enabled linkin a subnet topology.
 5. The method according to claim 1, wherein thefabric network is an InfiniBand network that includes a plurality ofsubnets.
 6. The method according to claim 1, wherein the fabric networkspecifies one or more policies for defining a link in a non-routablestate for observing errors.
 7. The method according to claim 1, where anautomated logic is used to determine whether the one or more errorsexceeds the configurable threshold.
 8. A non-transitory machine readablestorage medium having instructions stored thereon that when executedcause a system to perform the steps comprising: monitoring, via a daemonprocess on a first network node in the fabric network, a local port ofthe first network node for one or more errors, wherein the one or moreerrors are associated with a link that connects the first network nodeand a second network node in the fabric network, wherein the link is ina first state and is configured to transfer both normal data and subnetmanagement packet (SMP) data; in response to the one or more errorsexceeding a configurable threshold, performing, by the daemon process, adisabling operation on the link to put the link into a second state,wherein the link in the second state is configured to transfer only SMPdata; recording, by the daemon process, the state of the link in apersistent storage, wherein the recorded state is configured to preventthe link from being automatically enabled when the first network node isreset; and sending, via the link, one or more SMP messages to the subnetmanager, wherein the one or more management data messages indicate thatthe link is disabled; wherein the subnet manager, upon receiving the oneor more management data messages, operates to use the link to performfurther discovery in the second network node, invoke the daemon processto initiate a test on the link, determine that errors at the local portof the first network node have dropped below the configurable threshold,and change the link from the second state to the first state; andwherein the daemon process, in response to the state change of the linkfrom the second state to the first state, operates to remove therecorded state from the persistence storage when the first network nodeis reset.
 9. The non-transitory machine readable storage mediumaccording to claim 8, wherein the fabric network is an InfiniBandnetwork that includes a plurality of subnets.
 10. The non-transitorymachine readable storage medium according to claim 8, wherein thenetwork node operates to request the disabled link to be enabled. 11.The non-transitory machine readable storage medium according to claim 8,wherein the subnet manager observes the one or more errors using subnetmanagement packet (SMP) methods.
 12. The non-transitory machine readablestorage medium according to claim 8, wherein the subnet manager operatesto include the enabled link in a subnet topology.
 13. The non-transitorymachine readable storage medium according to claim 8, wherein the fabricnetwork specifies one or more policies for defining a link in anon-routable state for observing errors.
 14. A system for supportingautomatic disabling of degraded links in a network, comprising: acomputer one or more microprocessors; a fabric network executing on thecomputer, wherein the fabric network includes a first network node and asecond network node, wherein the first network node includes a daemonprocess configured to monitor a local port of the network node for oneor more errors associated with a link that connects the first networknode and the second network node, wherein the link is in a first stateand is configured to transfer both normal data and subnet managementpacket (SMP) data, perform a disabling operation on the link to put thelink into a second state, in response to the one or more errorsexceeding a configurable threshold, wherein the link in the second stateis configured to transfer only SMP data, and record the state of thelink in a persistent storage, wherein the recorded state is configuredto prevent the link from being automatically enabled when the networknode is reset; and a subnet manager in the fabric network, wherein thesubnet manager operates to receive, via the link, one or more managementdata messages from the first network node, wherein the one or moremanagement data messages indicate that the link is disabled; wherein themanaging node, upon receiving the one or more management data messages,operates to use the link to perform further discovery in the secondnetwork node, invoke the daemon process to initiate a test on the link,determine that errors at the local port of the first network node havedropped below the configurable threshold, and change the link from thesecond state to the first state; and wherein the daemon process, inresponse to the state change of the link from the second state to thefirst state, operates to remove the recorded state from the persistencestorage when the first network node is reset.
 15. The system accordingto claim 14, wherein the network node operates to request the disabledlink to be enabled.
 16. The system according to claim 14, wherein thesubnet manager operates to include the enabled link in a subnettopology.
 17. The system according to claim 14, wherein the fabricnetwork is an InfiniBand network that includes a plurality of subnets.18. The system according to claim 14, wherein the fabric networkspecifies one or more policies for defining a link in a non-routablestate for observing errors.
 19. The system according to claim 14, wherean automated logic is used to determine whether the one or more errorsexceeds the configurable threshold.